This invention relates to a system and method for logging and archiving network data traffic. Business and legal requirements may require monitoring of network data traffic, which may include data packets flowing across the network. For example, anti-terrorism laws may require an Internet Service Provider (ISP) to maintain logs of all Internet traffic of its customers for a prescribed time period. The goals of such laws are to assist law enforcement agencies to investigate potential terrorist activities, including planning and financing. Other goals may include investigating potential lawbreakers and thwarting child pornographers and other internet predators. Investigations into illicit behavior are often hampered because such log data is routinely deleted in the normal course of business. Furthermore, the value of the current log is limited due to the fact that it contains very basic metadata (data about data) and nothing about the data traffic payload. Corporations may use this data to help them better manage their networks and to identify anomalous or unwanted network traffic. This data, however, is subject to the same limitations as described above.
Storing the entire network traffic is technically feasible, but this approach would come at great cost in terms of storage and archival. In addition, the laws of some countries may prohibit inspection of people's data without court approval or other authorization on a case by case basis. Furthermore, even if the entire traffic data were retained, there is no method to efficiently and effectively search the data. In the US, legislation has been enacted and new legislation is proposed to permit limited surveillance in the form of logging. Such logging may keep the names of an ISP's customers and their IP addresses, the IP addresses of the sites to which they connected, and the dates and times of their connections. Because the goal is investigative, the paucity of data limits the value of the log. For example, if investigators were to have the entire network traffic available for inspection, including the payload, the quality of their data would improve significantly, thus aiding their investigation. However, this is not feasible, due to various laws prohibiting such surveillance. In corporate use, the cost associated with storing all network traffic may not be justifiable.
There is a need, therefore, for an improved method, article of manufacture, and apparatus for monitoring network traffic.